Skip to content

Adding 'invocation_id' to run-results as expected by OL#51916

Merged
potiuk merged 1 commit intoapache:mainfrom
amoghrajesh:fixing-dbt-tetss
Jun 19, 2025
Merged

Adding 'invocation_id' to run-results as expected by OL#51916
potiuk merged 1 commit intoapache:mainfrom
amoghrajesh:fixing-dbt-tetss

Conversation

@amoghrajesh
Copy link
Contributor

@amoghrajesh amoghrajesh commented Jun 19, 2025

Openlineage released 1.34.0:

> openlineage-integration-common==1.34.0
> openlineage-python==1.34.0
> openlineage_sql==1.34.0

This causes a breaking change in OpenLineage 1.34.0 where the DBT processor expects an invocation_id field in the run metadata, but the test data didn't include this field.

Added the invocation_id field to the metadata section of the test run_results.json file and assigned it equal to run_id:

{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v4.json",
    "dbt_version": "1.6.1",
    "invocation_id": "188471607"
  }
}

Fixes CI:

______ TestGenerateOpenLineageEventsFromDbtCloudRun.test_generate_events _______
[gw2] linux -- Python 3.9.23 /usr/local/bin/python3
providers/dbt/cloud/tests/unit/dbt/cloud/utils/test_openlineage.py:187: in test_generate_events
    generate_openlineage_events_from_dbt_cloud_run(mock_operator, task_instance=mock_task_instance)
providers/common/compat/src/airflow/providers/common/compat/openlineage/check.py:100: in wrapper
    return func(*args, **kwargs)
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:192: in generate_openlineage_events_from_dbt_cloud_run
    events = processor.parse().events()
/usr/local/lib/python3.9/site-packages/openlineage/common/provider/dbt/processor.py:201: in parse
    events += self.parse_execution(context, nodes)
/usr/local/lib/python3.9/site-packages/openlineage/common/provider/dbt/processor.py:299: in parse_execution
    self.get_run(run_id),
/usr/local/lib/python3.9/site-packages/openlineage/common/provider/dbt/processor.py:685: in get_run
    "dbt_run": self.dbt_run_run_facet(),
/usr/local/lib/python3.9/site-packages/openlineage/common/provider/dbt/processor.py:704: in dbt_run_run_facet
    return DbtRunRunFacet(invocation_id=self.run_metadata["invocation_id"])  # type: ignore[index]
E   KeyError: 'invocation_id'

Example failure: https://github.com/apache/airflow/actions/runs/15748852234

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh amoghrajesh requested a review from josh-fell as a code owner June 19, 2025 05:59
@potiuk potiuk merged commit 398ba86 into apache:main Jun 19, 2025
55 checks passed
@kacpermuda
Copy link
Contributor

kacpermuda commented Jun 19, 2025

LGTM, It's caused by OpenLineage/OpenLineage#3738, now we're including invocation_id in DBT facets. @dolfinus am I assuming correctly that it's always in the metadata returned by DBT?

@dolfinus dolfinus mentioned this pull request Jun 19, 2025
Merged
8 tasks
@dolfinus
Copy link
Contributor

dolfinus commented Jun 19, 2025

According to https://github.com/dbt-labs/dbt-core/pull/4340/files#diff-33c94e40283afdfeba411a98c0628fc610c287e41c3477d88559f1e0244259c3R317, invocation_id is a part of manifest since dbt-core 1.0.0.
But in manifest schema https://github.com/dbt-labs/dbt-core/blob/main/schemas/dbt/manifest/v12.json this field is marked as nullable. Let's make it optional OpenLineage/OpenLineage#3796

RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Jun 21, 2025
@amoghrajesh @RoyLee1224
Adding 'invocation_id' to run-results as expected by OL (apache#51916)
25e58f4
@eladkal eladkal mentioned this pull request Jul 3, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

@josh-fell josh-fell Awaiting requested review from josh-fell josh-fell is a code owner

@mobuchowski mobuchowski Awaiting requested review from mobuchowski

@eladkal eladkal Awaiting requested review from eladkal

Assignees

No one assigned

Labels

area:providers provider:dbt-cloud

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@amoghrajesh @kacpermuda @dolfinus @potiuk